Your SlideShare is downloading. ×
An Introduction to AMQP with Code Samples
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

An Introduction to AMQP with Code Samples

3,387
views

Published on

An introduction to AMQP with code samples by StormMQ at SkillsMatter, London on the 26th October.

An introduction to AMQP with code samples by StormMQ at SkillsMatter, London on the 26th October.

Published in: Technology

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,387
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
85
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. What’s Messaging? An Introduction
  • 2. History of Message Queuing Manual Telegraphy Machine Assisted Telegraphy 1920s1911 - 192019th Century 1900s Telegrams sent using “Store and Forward” 1900 1930s
  • 3. History of Message Queuing Electronic Telegraphy 1950s1940s1950s1940s Electronic Telegram Machines, eg Plan 55-A 1948 IBM M ne Assisted Telegraphy 1920s1920 1930s
  • 4. History of Message Queuing Telcos UseElectronic Telegraphy 1950s1940s 1960s IBM System/360 with BTAM & QTAM Message Switching 1964 First Electronic Mail Solutions 1965 Banking Users 1970s IBM TCAM which is the first true solution Retired 1990! 1971
  • 5. History of Message Queuing l Financial Trading UsersBanking Users FMCG & Utilities 1980s 1990s Growth of SMTP Origins of Tibco in Stock Price Messaging 1980s IBM Launch MQSeries (now WebsphereMQ) 1992 1970s IBM TCAM which is the first true solution Retired 1990! 1971
  • 6. History of Message Queuing l FMCG & Utilities 1990s MQSeries phereMQ) 92 Corporates Large Websites YOU Noughties Today Sun Release Java JMS, Reinvigorating Enterprise Messaging 2001 AMQP Working Group Formed by Investment Banks 2006 Cloud Enables and Drives StormMQ Adoption 2009
  • 7. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  • 8. The Integration Tag Soup WS –File Transfer !"#$%&'(#)#*#+%+ Message Queuing SOAP EMail SMTP FTP RSYNC NTFS !','- !).$%&'/$.0%&1$%+ 234%5)6!%$7%$ SOA Message Orientated Middleware JMS MQ MQSeries AMQP Amazon SQS Tibco StormMQ RabbitMQ HTTP TCP/IP DCE / RPC CORBA DCOM REST RMI XML-RPC .NET Remoting Remote Procedure Call
  • 9. Under Examination, though File Transfer !"#$%&'(#)#*#+%+ Remote Procedure Call Don’tWork in the Cloud
  • 10. Under Examination, though Deployment Intimate System Knowledge Configuration Admin File Transfer !"#$%&'(#)#*#+%+ Remote Procedure Call Don’tWork in the Cloud Scaling
  • 11. Ideal: Message Queuing Systems are “loosely-coupled” Message Queuing
  • 12. Ideal: Message Queuing Systems are “loosely-coupled” Message Queuing
  • 13. Ideal: Message Queuing Systems are “loosely-coupled” Message Queuing Systems don’t know each other
  • 14. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards?
  • 15. Why Use it: Loose Coupling Billing Catalogue Shipping S-a-a-S Inventory How do we connect them, without one outage or system change taking everything down like a pack of cards? Message Queuing lets Systems and Components exchange data, events, commands and actions with one another with no explicit knowledge or need for them to be online
  • 16. It should be ideal for the cloud
  • 17. It should be ideal for the cloud
  • 18. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc) Platform Restricted BespokeProprietary
  • 19. ! ! ! Until today, messaging is … (ZeroMQ) (SQS uses HTTP) (MSMQ uses COM) (memcached) (Java JMS uses Source) (MQSeries / WebsphereMQ) (Talarian, Rendezvous, etc)
  • 20. Why is it Hell? You need more staff And moneyYou need bridge technology
  • 21. Why is it Hell? You need more staff And moneyYou need bridge technology
  • 22. However, there’s a solution
  • 23. A-MQPAdvanced Message Queue Protocol However, there’s a solution
  • 24. However, AMQP fixes this A common wire-level binary format and protocol An explicit definition of a server (aka broker)’s semantics Open Means Interoperable
  • 25. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable
  • 26. That is good … “AMQP will be to Messaging what HTTP was to theWeb” MRG Clients run on any Platform Vendors are Interoperable 64K
  • 27. From Anywhere to Anywhere
  • 28. From Anywhere to Anywhere
  • 29. From Anywhere to Anywhere
  • 30. From Anywhere to Anywhere
  • 31. From Anywhere to Anywhere
  • 32. Quick Recap For Beer The fifth male member of the A-Team, Frankie, was played by Question Message Queuing connects systems and components. Is it ideal for the cloud? Does Loose-Coupling make individual systems more likely to suffer outages? Yes No AMQP is Open. This makes it suitable for programming in C and Javascript?
  • 33. EddieVelez Quick Recap ✓ The fifth male member of the A-Team, Frankie, was played by Question Message Queuing connects systems and components. Is it ideal for the cloud? Does Loose-Coupling make individual systems more likely to suffer outages? Yes No AMQP is Open. This makes it suitable for programming in C and Javascript? ! ✓
  • 34. Common Terms and Jargon Jargon
  • 35. Messaging: Which Jargon? EMail (SMTP, POP3, IMAP) VoIP (VoiceMail, XMPP) Texting (SMS) Instant Messaging (ICQ, MSN, Jabber) Twitter Enterprise Service Bus (ESB) Dynamic OO Languages (eg Ruby) Message Queuing (MQ) Message Queuing (MQ) What do we mean by Messaging?
  • 36. Essential Terms Message Queue Behaves like a Queue Is First-In First-Put (FIFO) Elements are Messages
  • 37. Essential Terms Message Queue What’s a Message? Envelope Payload Header
  • 38. Essential Terms *Strictly speaking a receiver polls for messages a consumer has messages pushed ≣ ≊Send Publish Enqueue ≣ ≊ DequeueConsume*Receive
  • 39. So how can you use it? Concepts Useful Book: Enterprise Integration Patterns Hohpe &Woolf
  • 40. Store and Forward Billing “Could you send me a duplicate of my bill please” REST
  • 41. Store and Forward Billing “Could you send me a duplicate of my bill please” REST !
  • 42. Store and Forward Billing “Could you send me a duplicate of my bill please” REST StormMQ Messaging Billing !
  • 43. Store and Forward Billing “Could you send me a duplicate of my bill please” REST StormMQ Messaging Billing ! !
  • 44. Store and Forward Billing “Could you send me a duplicate of my bill please” REST StormMQ Messaging Billing !
  • 45. Fire and Forget Shipping “Too many orders.The website is running like a dog.” SQL Push SQL Pull
  • 46. Fire and Forget Shipping “Too many orders.The website is running like a dog.” SQL Push SQL Pull StormMQ Messaging Shipping
  • 47. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory File Transfer ETL
  • 48. One-To-Many “There’s a new widget in the Catalogue: Tell all the systems.” Catalogue Shipping S-a-a-S Inventory StormMQ Messaging
  • 49. Publish-Subscribe (“Topics”) “Shipments Sent, Delivered and Returned” Shipping S-a-a-S Inventory Billing AnySent Sent or Returned
  • 50. StormMQ Messaging Publish-Subscribe (“Topics”) “Shipments Sent, Delivered and Returned” Shipping S-a-a-S Inventory Billing AnySent Sent or Returned
  • 51. Round-Robin Billing “How do we easily scale a massive batch job like Billing?” EMail BillGenerate
  • 52. StormMQ Messaging Round-Robin Billing “How do we easily scale a massive batch job like Billing?” EMail BillGenerate
  • 53. Getting Technical Delving into AMQP
  • 54. AMQP Client AMQP ServerTCP / IP Network Connection Virtual Host Connections and Channels
  • 55. Connection AMQP Client AMQP ServerTCP / IP Network Connection Virtual Host Connections and Channels TLS “Shielding” Channels Each Channel is Independent: Effectively, aVirtual Connection
  • 56. Basic AMQP: Connections Basic AMQP: Connections Open a Connection to aVirtual Host Open a Channel Send a Message Receive a Message Close Channel Close Connection You only need one channel! " # $ % & ' How does this look in Code?
  • 57. // The API is styled similarly to Functionally Orientated Programming (FOP) // It makes extensive use of Constructor Injection and anonymous classes modelling 'blocks' // Closes are handled implicitly by design with exceptions thrown, ie try-finally is internal to the API public void demonstrateConnection() { final AmqpConnectionFactory amqpConnectionFactory = amqpConnectionFactoryWhichVerifiesStormMQ(“mycompany”, “mysystem”, “development”, “user”, “password”); // Creates and closes a connection amqpConnectionFactory . useConnection ( // Creates and closes a channel on a connection new ChannelCreatingConnectionUser ( // Adapts a channel to make it convenient to use new ConvenientChannelCreatingConnectionUser ( new ConvenientChannelUser() ( public void use(final @NotNull ConvenientChannel convenientChannel) { // Send a message, or, // Receive a message (or both) } ); ) ) ) }
  • 58. // Sending a Message public void use(final @NotNull ConvenientChannel convenientChannel) { // ConvenientChannel has static convenience methods to create messageHeaders (meta-data) // Use them to signify a message's content’s MIME type or an unique id // Use meta data sparingly; think of it like email headers. More later final BasicProperties messageHeaders = emptyBasicProperties(); // All messages are byte arrays final byte[] messageBody = "HelloWorld" . getBytes( forName("UTF-8") ); // Send the message final String routingKey = "usually the destination message queue"; final boolean mandatory = false; final boolean immediate = false; convenientChannel . basicPublish( "", routingKey, mandatory, immediate, messageHeaders, messageBody ); } /* Notes * Mandatory (mandatory = true) messages are returned unless routed to a queue * Immediate (mandatory = true) messages are returned unless there is another channel consuming */
  • 59. // Receiving a Message using Polling public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean noAck = false; convenientChannel . basicGet( "queue", noAck, DoNothingWhenMessageNotReceived, new MessageReceived() { public void messageReceived(final @NotNull GetResponse message) { final byte[] messageBody = message . getBody(); final BasicProperties messageHeaders = message . getProps(); // Acknowledge successfully received messages if noAck = false // If not acknowledged, they are eventually placed back on the queue for re-delivery convenientChannel . basicAck( message ); } }); } /* Notes * DoNothingWhenMessageNotReceived is a singleton of MesageNotReceived * Implement DoNothing to do something else, eg for debugging * Polling is thread-safe but inefficient; using consumers (“push”) for more efficiency… */
  • 60. // Consuming Messages using Push public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean noAck = true; convenientChannel . basicConsume( "my queue", noAck, new ConvenientConsumer() { // Consumers run on a different thread public void handleDelivery( final String consumerTag, final Envelope envelope, final BasicProperties messageHeaders, final byte[] messageBody ) { final byte[] messageBody = message . getBody(); final BasicProperties messageHeaders = message . getProps(); // No need to acknowledge message as noAck = true } }); } /* Notes * Consumers only exist as long as a channel is open * Add a threading primitive (eg volatile boolean) and reference it at * * It is possible to use multiple channels with several consumers… */ *
  • 61. ! ✓ Channels Complex Applications Consume on several threads Send on several threads Why Use More than One Channel? Recommendation: Keep it Simple! Simple Applications Send on a queue Receive on another Transactional Applications NeedTransactions on a queue Don’t NeedTransactions on another ✓
  • 62. Exchanges Route Messages Exchanges route Messages to Message Queues Exchange MQ A MQ B
  • 63. Exchanges Route Messages Exchanges route Messages to Message Queues You send messages to an exchange, not a message queue Exchange MQ A MQ B
  • 64. How do Exchanges Route? Exchange MQ A The Exchange finds a Binding matching the Routing Key # A Binding connects a Routing Key to one or more Message Queues $ Every sent message has a Routing Key " The Exchange delivers the Message to the Message Queue % A message queue can be bound more than once to one or more exchanges ( “string”
  • 65. Types of Exchange More rarely used exchange types include amq.headers and extensions Message Queue Name direct Like a ‘Map’:All MQs bound with the routing key receive copies of the message “” (blank) amq.direct fanout Empty String All MQs bound to the Exchange receive copies of the message amq.fanout topic Dotted Bindings use globbing expressions (wildcards) to route messages to MQs amq.topic A message queue can be bound more than once to an exchange; A message queue can be bound to more than one exchange But a message queue will only receive a message sent once Routing Key Routing Behaviour Default Definitions Point – to – Point One – to – Many Publish – SubscribeTypical Use
  • 66. // Declaring Exchanges public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean passivelyDeclare = false; final boolean durable = true; final boolean autoDelete = false; // Just change the ExchangeType to one of direct, fanout or topic convenientChannel . exchangeDeclare( "my exchange A", direct, passivelyDeclare, durable, autoDelete ); convenientChannel . exchangeDeclare( "my exchange B", fanout, passivelyDeclare, durable, autoDelete ); convenientChannel . exchangeDeclare( "my exchange C", topic, passivelyDeclare, durable, autoDelete ); // A passive declaration asserts that an exchange exists, causing a channel exception if it does not convenientChannel . exchangeDeclare( "my exchange A", direct, true, durable, autoDelete ); } /* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as exchanges should be long-lived * Declaring an exchange twice is not an error; if parameters differ, new values are used if possible * Passive declaration is interesting but typical deployment practices mitigate its usefulness */
  • 67. // Declaring and Binding Message Queues public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean passivelyDeclare = false; final boolean durable = true; final boolean autoDelete = false; final boolean exclusive = false; // A declared queue is bound to the default ("") exchange with the routing key the queue name ("my queue") convenientChannel . queueDeclare( "my queue", passivelyDeclare, durable, exclusive, autoDelete ); // A message queue can be bound to another exchange with whatever key you choose convenientChannel . queueBind( "my queue", "another exchange", "some routing key" ); // When binding to a fanout exchange the routing key is the empty string ("") convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.fanout", "" ); // When binding to a topic exchange, set the routing key as a wildcard // Publish messages with specific routing keys eg "BONDS.USD.FTSE", "BONDS.USD.NYSE", "BONDS.GBP.FTSE" convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.topic", "BONDS.USD.*" ); } /* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as message queues should be long-lived * Declaring a message queue twice is not an error; if parameters differ, new values are used if possible * Exclusive message queues are for advanced usages and often aren’t needed */
  • 68. AMQP: Introducing Users but StormMQ enforces that the same Users and Permissions are in all of a System’s Environments Passwords are different Passwords are securely generated by StormMQ and are 512 bit keys Configuration Managers can prevent revelation of passwords to different system users, eg developers vs sysadmins Aim: Separation of Environments A Connection can use several authentication mechanisms which are perVirtual Host but Commonest is User - Password andand Aim: Separation of Responsibility Users are intended to be system accounts (robots), not sysadmins Users have read, write and create permissions using a wildcard syntax: important to name queues, etc well
  • 69. // Example of create-system.json used with API call stormmq-create-system { "companyName" : "usuallyYourUserName", "systemName" : "anythingAlphabetic", "environments" : [{ "environmentName" : "development", // Different StormMQ clusters have different characteristics (and costs) "clusterName" : "free-1", // These are the StormMQ user accounts to which passwords for AMQP users (below) can be revealed for this environment (development) "permittedStormMQUserNames" : [ "developerPeter", "developerJames", "developerJohn" ] }], // User permissions operate on Message Queues and Exchanges "amqpUserPermissions" : { "queueReader" : { "create" : "^$", "read" : ".*", "write" : "^$" }, "queueWriter" : { "create" : "^$", "read" : "^$", "write" : ".*" }, "queueCreator" : { "create" : ".*", "read" : "^$", "write" : "^$" }, "onlyProcessingObjects" : { "create" : "^processing.*", "read" : "^processing.*", "write" : "^processing.*" } } }
  • 70. Configuring AMQP “*Our REST API has Java, Ruby, PHP and .NET bindings” AMQP StormMQ ✓ ActiveMessage Queues Exchanges Bindings Users User Permissions Entire Systems ! Extra ! Extra ! Extra ✓ Active ✓ Active ✓ REST ✓ REST ✓ REST* Programmatic Config ✓ Same ✓ Same ✓ Same
  • 71. AMQP: Getting Statistics “*Password revelation is restricted to specific individuals” AMQP StormMQ Message Queues Exchanges Bindings Generated Passwords Usage Patterns Entire Systems ! Extra ! Extra ! Extra ✓ REST ✓ REST ✓ REST Programmatic Info ! Extra ✓ REST ! Extra ✓ REST* ! Extra ✓ REST
  • 72. // Getting Statistics from StormMQ using the REST API in Java // If you need to use proxies or experience the pain of Java SSL choose a different method in ApiConfiguration final ApiConfiguration apiConfiguration = directConnectionIfHavingProblemsWithSslVerification(); // If your Magic Secret Key is in an unusual place choose a different method in Api final Api api = apiByGuessingUsingOsConfigurationFiles( apiConfiguration, stormMqUserName ); final List<AmqpQueue> amqpQueues = api . listQueues( companyName, systemName, environmentName ); final List<AmqpExchange> amqpExchanges = api . listExchanges( companyName, systemName, environmentName ); final List<AmqpBinding> amqpBindings = api . listBindings( companyName, systemName, environmentName ); final Map<AmqpUserName, AmqpUserPassword> passwords = api . listAmqpUserNamesAndPasswords( companyName, systemName, environmentName ); final CompanyDetails companyDetails = api . describeCompany( companyName ); // Finding “what we’ve got” final List<CompanyName> companyNames = api . listCompanies(); final List<ClusterName> clusterNames = api . listClusters( companyName ); final List<SystemName> systemNames = api . listSystems( companyName ); final SystemDetails systemDetails = api . describeSystem( companyName, systemName ); final Set<Environment> environments = systemDetails . getEnvironments(); /* Notes * Also consider the command-line tools stormmq-list-queues, stormmq-list-exchanges and stormmq-list-bindings * Further examples are in com.stormmq.api.Example (part of the url signer jar) */
  • 73. Virtual Hosts Virtual Hosts: Think Apache! Virtual Hosts isolate Message Queues Virtual Hosts isolate User Groups* Virtual Hosts isolate Exchanges* Virtual Hosts isolate Bindings* Virtual Hosts isolate Virtual Hosts: StormMQ Virtual Hosts isolateYour Company Virtual Hosts isolateYour Systems Virtual Hosts isolateYour Environments Enable Configuration Management Virtual Hosts stop Data Accidents Widgets Ltd / Invoicing System / / Testing Development Production
  • 74. Message Properties Server Understood Good Practice “This is just the tip of messaging best practice with AMQP” Persistent = 1 Priority = 0 Timestamp = 4568965345 MessageId = 567A-GH-20100709 Priority Queues are per-Message Control life of a Message Persistence is per-Message Timestamps expire Messages Completely Optional MIME Content-Type Unique MessageId Schema &Version
  • 75. // Message Properties: Examples of Good Practice // Being explicit is good practice final BasicProperties messageHeaders = new BasicProperties(); {{ DeliveryMode . NonPersistent . setOn( this ); Priority . Zero . setOn( this ); setMessageId( "My unique ID, eg a GUID or ascending value" ); // Explicit MIME type including charset; use application/octect-stream for trully binary data setContentType("application/json; charset=utf-8"); // Use compression for larger textual data; use the same values as HTTP headers; use "identity" for none setContentEncoding( "gzip" ); // Timestamp each message setTimestamp( new Date( System . currentTimeMillis() ) ); // Schema and version, perhaps using a XML namespace definition setType( "syslog-message_1.0.4" ); // Optional: Use an expiration to discard old messages that are stale (milliseconds, example is 5 minutes) setExpiration( Integer . toString( 5 * 60 * 1000) ); // Optional: Identify the sender for later debugging setAppId( "sending application name" ); setUserId( "role of sender, or, user of machine" ); // Optional: Identify domain or environment (consumer could error if working in different one) setClusterId( "production" ); }};
  • 76. And there is more! Transactions More Message Properties Custom Message Properties Immediate Delivery Additional Exchange Types Auto Deletion Fine-Grained User Permissions Queue Purging QoS
  • 77. Useful Book: Enterprise Integration Patterns Hohpe &Woolf