• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content

Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

An Introduction to AMQP with Code Samples

on

  • 3,515 views

An introduction to AMQP with code samples by StormMQ at SkillsMatter, London on the 26th October.

An introduction to AMQP with code samples by StormMQ at SkillsMatter, London on the 26th October.

Statistics

Views

Total Views
3,515
Views on SlideShare
3,404
Embed Views
111

Actions

Likes
4
Downloads
80
Comments
0

2 Embeds 111

http://blog.stormmq.com 110
http://static.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    An Introduction to AMQP with Code Samples An Introduction to AMQP with Code Samples Presentation Transcript

    • What’s Messaging? An Introduction
    • History of Message Queuing Telegrams sent using “Store and Forward” 1900 19th Century 1900s 1911 - 1920 1920s 1930s Manual Telegraphy Machine Assisted Telegraphy
    • History of Message Queuing IBM Electronic Telegram Machines, eg Plan 55-A M 1948 1920 1920s 1930s 1940s 1950s 1940s 1950s ne Assisted Telegraphy Electronic Telegraphy
    • History of Message Queuing IBM System/360 with IBM TCAM which is First Electronic Mail BTAM & QTAM the first true solution Solutions Message Switching Retired 1990! 1964 1965 1971 1940s 1950s 1960s 1970s Electronic Telegraphy Telcos Use Banking Users
    • History of Message Queuing IBM TCAM which is Growth of SMTP l IBM Launch MQSeries the first true solution Origins of Tibco in (now WebsphereMQ) Retired 1990! Stock Price Messaging 1971 1980s 1992 1970s 1980s 1990s Banking Users Financial Trading Users FMCG & Utilities
    • History of Message Queuing Sun Release Java JMS, AMQP Working Cloud Enables and l MQSeries Reinvigorating Group Formed by Drives StormMQ phereMQ) Enterprise Messaging Investment Banks Adoption 92 2001 2006 2009 1990s Noughties Today FMCG & Utilities Corporates Large Websites YOU
    • The Integration Tag Soup Remote Procedure Call !"#$%&'(#)#*#+%+ EMail !','- StormMQ JMS CORBA MQSeries Amazon SQS RMI RSYNC SOA XML-RPC HTTP Message Message Queuing NTFS MQ Orientated SOAP FTP DCOM Tibco Middleware .NET Remoting SMTP REST !).$%&'/$.0%&1$%+ AMQP DCE / RPC RabbitMQ TCP/IP File Transfer WS – 234%5)6!%$7%$
    • The Integration Tag Soup Remote Procedure Call !"#$%&'(#)#*#+%+ EMail !','- StormMQ JMS CORBA MQSeries Amazon SQS RMI RSYNC SOA XML-RPC HTTP Message Message Queuing NTFS Orientated MQ SOAP FTP DCOM Tibco Middleware .NET Remoting SMTP REST AMQP !).$%&'/$.0%&1$%+ DCE / RPC RabbitMQ TCP/IP File Transfer WS – 234%5)6!%$7%$
    • Under Examination, though !"#$%&'(#)#*#+%+ Remote Procedure Call File Transfer Don’t Work in the Cloud
    • Under Examination, though Admin Deployment !"#$%&'(#)#*#+%+ Configuration Scaling Remote Procedure Call Intimate System Knowledge File Transfer Don’t Work in the Cloud
    • Ideal: Message Queuing Message Queuing Systems are “loosely-coupled”
    • Ideal: Message Queuing Message Queuing Systems are “loosely-coupled”
    • Ideal: Message Queuing Message Queuing Systems don’t know each other Systems are “loosely-coupled”
    • Why Use it: Loose Coupling Catalogue Billing How do we connect them, without one outage or system change taking everything down like a pack of cards? Shipping S-a-a-S Inventory
    • Why Use it: Loose Coupling Catalogue Billing Message Queuing lets Systems and Components exchange How do we connect them, without one outage or system data, events, commands and actions with one another change taking everything down like a pack of cards? with no explicit knowledge or need for them to be online Shipping S-a-a-S Inventory
    • It should be ideal for the cloud
    • It should be ideal for the cloud
    • Until today, messaging is … !! Proprietary Platform Restricted Bespoke (MQSeries / WebsphereMQ) ! (MSMQ uses COM) (ZeroMQ) (Java JMS uses Source) (Talarian, Rendezvous, etc) (memcached) (SQS uses HTTP)
    • Until today, messaging is … !! (MQSeries / WebsphereMQ) ! (MSMQ uses COM) (ZeroMQ) (Java JMS uses Source) (Talarian, Rendezvous, etc) (memcached) (SQS uses HTTP)
    • Why is it Hell? You need bridge technology You need more staff And money
    • Why is it Hell? You need bridge technology You need more staff And money
    • However, there’s a solution
    • However, there’s a solution A-MQP Advanced Message Queue Protocol
    • However, AMQP fixes this Open Means A common wire-level binary format and protocol An explicit definition of a server (aka broker)’s semantics Interoperable
    • That is good … Clients run on any Platform Vendors are Interoperable MRG “AMQP will be to Messaging what HTTP was to the Web”
    • That is good … Clients run on any Platform Vendors are Interoperable 64K MRG “AMQP will be to Messaging what HTTP was to the Web”
    • From Anywhere to Anywhere
    • From Anywhere to Anywhere
    • From Anywhere to Anywhere
    • From Anywhere to Anywhere
    • From Anywhere to Anywhere
    • Quick Recap Question Yes No Message Queuing connects systems and components. Is it ideal for the cloud? Does Loose-Coupling make individual systems more likely to suffer outages? AMQP is Open. This makes it suitable for programming in C and Javascript? The fifth male member of the A-Team, Frankie, was played by For Beer
    • Quick Recap Question Yes No Message Queuing connects systems and components. Is it ideal for the cloud? ✓ Does Loose-Coupling make individual systems more likely to suffer outages? ! AMQP is Open. This makes it suitable for programming in C and Javascript? ✓ The fifth male member of the A-Team, Frankie, was played by Eddie Velez
    • Common Terms and Jargon Jargon
    • Messaging: Which Jargon? Dynamic OO Languages (eg Ruby) Instant Messaging Message Queuing (ICQ, MSN, Jabber) (MQ) What do we EMail VoIP (SMTP, POP3, IMAP) mean by (VoiceMail, XMPP) Messaging? Texting Enterprise Service Bus (SMS) (ESB) Twitter
    • Essential Terms Message Queue Behaves like a Queue Is First-In First-Put (FIFO) Elements are Messages
    • Essential Terms Message Queue What’s a Message? Header Envelope Payload
    • Essential Terms Send ≣ Publish ≊ Enqueue Receive ≣ Consume* ≊ Dequeue *Strictly speaking a receiver polls for messages a consumer has messages pushed
    • So how can you use it? Concepts Useful Book: Enterprise Integration Patterns Hohpe & Woolf
    • Store and Forward REST Billing “Could you send me a duplicate of my bill please”
    • Store and Forward REST !Billing “Could you send me a duplicate of my bill please”
    • Store and Forward REST StormMQ Messaging !Billing Billing “Could you send me a duplicate of my bill please”
    • Store and Forward ! ! REST Billing StormMQ Messaging Billing “Could you send me a duplicate of my bill please”
    • Store and Forward REST StormMQ Messaging !Billing Billing “Could you send me a duplicate of my bill please”
    • Fire and Forget Shipping SQL Push SQL Pull “Too many orders.The website is running like a dog.”
    • Fire and Forget Shipping SQL Push SQL Pull StormMQ Shipping Messaging “Too many orders.The website is running like a dog.”
    • One-To-Many File Transfer Shipping Catalogue ETL S-a-a-S Inventory “There’s a new widget in the Catalogue: Tell all the systems.”
    • One-To-Many Shipping Catalogue S-a-a-S Inventory StormMQ Messaging “There’s a new widget in the Catalogue: Tell all the systems.”
    • Publish-Subscribe (“Topics”) Shipping Sent or Sent Any Returned S-a-a-S Inventory Billing “Shipments Sent, Delivered and Returned”
    • Publish-Subscribe (“Topics”) Shipping StormMQ Messaging Sent or Sent Any Returned S-a-a-S Inventory Billing “Shipments Sent, Delivered and Returned”
    • Round-Robin Generate EMail Bill Billing “How do we easily scale a massive batch job like Billing?”
    • Round-Robin Generate EMail Bill Billing StormMQ Messaging “How do we easily scale a massive batch job like Billing?”
    • Getting Technical Delving into AMQP
    • Connections and Channels Connection Virtual Host AMQP Client TCP / IP Network AMQP Server
    • Connections and Channels Channels TLS “Shielding” Connection Each Channel is Independent: Effectively, a Virtual Connection Connection Virtual Host AMQP Client TCP / IP Network AMQP Server
    • Basic AMQP: Connections Basic AMQP: Connections " Open a Connection to a Virtual Host # Open a Channel How does this look $ Send a Message % Receive a Message & Close Channel in Code? ' Close Connection You only need one channel!
    • / The API is styled similarly to Functionally Orientated Programming (FOP) / / It makes extensive use of Constructor Injection and anonymous classes modelling 'blocks' / / Closes are handled implicitly by design with exceptions thrown, ie try-finally is internal to the API / public void demonstrateConnection() { final AmqpConnectionFactory amqpConnectionFactory = amqpConnectionFactoryWhichVerifiesStormMQ(“mycompany”, “mysystem”, “development”, “user”, “password”); / Creates and closes a connection / amqpConnectionFactory . useConnection ( / Creates and closes a channel on a connection / new ChannelCreatingConnectionUser ( / Adapts a channel to make it convenient to use / new ConvenientChannelCreatingConnectionUser ( new ConvenientChannelUser() ( public void use(final @NotNull ConvenientChannel convenientChannel) { / Send a message, or, / / Receive a message (or both) / } ) ) ) ); }
    • / Sending a Message / public void use(final @NotNull ConvenientChannel convenientChannel) { / ConvenientChannel has static convenience methods to create messageHeaders (meta-data) / / Use them to signify a message's content’s MIME type or an unique id / / Use meta data sparingly; think of it like email headers. More later / final BasicProperties messageHeaders = emptyBasicProperties(); / All messages are byte arrays / final byte[] messageBody = "HelloWorld" . getBytes( forName("UTF-8") ); / Send the message / final String routingKey = "usually the destination message queue"; final boolean mandatory = false; final boolean immediate = false; convenientChannel . basicPublish( "", routingKey, mandatory, immediate, messageHeaders, messageBody ); } /* Notes * Mandatory (mandatory = true) messages are returned unless routed to a queue * Immediate (mandatory = true) messages are returned unless there is another channel consuming */
    • / Receiving a Message using Polling / public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean noAck = false; convenientChannel . basicGet( "queue", noAck, DoNothingWhenMessageNotReceived, new MessageReceived() { public void messageReceived(final @NotNull GetResponse message) { final byte[] messageBody = message . getBody(); final BasicProperties messageHeaders = message . getProps(); / Acknowledge successfully received messages if noAck = false / / If not acknowledged, they are eventually placed back on the queue for re-delivery / } convenientChannel . basicAck( message ); }); } /* Notes * DoNothingWhenMessageNotReceived is a singleton of MesageNotReceived * Implement DoNothing to do something else, eg for debugging * Polling is thread-safe but inefficient; using consumers (“push”) for more efficiency… */
    • / Consuming Messages using Push / public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean noAck = true; convenientChannel . basicConsume( "my queue", noAck, new ConvenientConsumer() { / Consumers run on a different thread / public void handleDelivery( final String consumerTag, final Envelope envelope, final BasicProperties messageHeaders, final byte[] messageBody ) { final byte[] messageBody = message . getBody(); final BasicProperties messageHeaders = message . getProps(); / No need to acknowledge message as noAck = true / } }); * } /* Notes * Consumers only exist as long as a channel is open * Add a threading primitive (eg volatile boolean) and reference it at * * It is possible to use multiple channels with several consumers… */
    • Channels Why Use More than One Channel? Simple Applications Send on a queue Receive on another ! Transactional Applications Need Transactions on a queue Don’t Need Transactions on another ✓ ✓ Complex Applications Consume on several threads Send on several threads Recommendation: Keep it Simple!
    • Exchanges Route Messages Exchanges route Messages to Message Queues A MQ Exchange B MQ
    • Exchanges Route Messages Exchanges route Messages to Message Queues A MQ Exchange B MQ You send messages to an exchange, not a message queue
    • How do Exchanges Route? Every sent message has a A Binding connects a Routing Key " $ Routing Key to one or more Message Queues “string” A The Exchange finds a Binding # matching the Routing Key ( A message queue can be bound more than once to one or more exchanges MQ The Exchange delivers the % Message to the Message Queue Exchange
    • Types of Exchange direct fanout topic Typical Use Point – to – Point One – to – Many Publish – Subscribe Routing Key Message Queue Name Empty String Dotted Like a ‘Map’: All MQs All MQs bound to the Bindings use Routing bound with the Exchange receive globbing expressions Behaviour routing key receive copies of the message (wildcards) to route copies of the message messages to MQs Default “” (blank) amq.fanout amq.topic Definitions amq.direct A message queue can be bound more than once to an exchange; A message queue can be bound to more than one exchange But a message queue will only receive a message sent once More rarely used exchange types include amq.headers and extensions
    • / Declaring Exchanges / public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean passivelyDeclare = false; final boolean durable = true; final boolean autoDelete = false; / Just change the ExchangeType to one of direct, fanout or topic / convenientChannel . exchangeDeclare( "my exchange A", direct, passivelyDeclare, durable, autoDelete ); convenientChannel . exchangeDeclare( "my exchange B", fanout, passivelyDeclare, durable, autoDelete ); convenientChannel . exchangeDeclare( "my exchange C", topic, passivelyDeclare, durable, autoDelete ); / A passive declaration asserts that an exchange exists, causing a channel exception if it does not / convenientChannel . exchangeDeclare( "my exchange A", direct, true, durable, autoDelete ); } /* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as exchanges should be long-lived * Declaring an exchange twice is not an error; if parameters differ, new values are used if possible * Passive declaration is interesting but typical deployment practices mitigate its usefulness */
    • / Declaring and Binding Message Queues / public void use(final @NotNull ConvenientChannel convenientChannel) { final boolean passivelyDeclare = false; final boolean durable = true; final boolean autoDelete = false; final boolean exclusive = false; / A declared queue is bound to the default ("") exchange with the routing key the queue name ("my queue") / convenientChannel . queueDeclare( "my queue", passivelyDeclare, durable, exclusive, autoDelete ); / A message queue can be bound to another exchange with whatever key you choose / convenientChannel . queueBind( "my queue", "another exchange", "some routing key" ); / When binding to a fanout exchange the routing key is the empty string ("") / convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.fanout", "" ); / When binding to a topic exchange, set the routing key as a wildcard / / Publish messages with specific routing keys eg "BONDS.USD.FTSE", "BONDS.USD.NYSE", "BONDS.GBP.FTSE" / convenientChannel . queueBind( "my queue", "a fanout exchange, eg amq.topic", "BONDS.USD.*" ); } /* Notes * Always use durable = true with StormMQ as we are a hosted service * Nearly always use autoDelete = false as message queues should be long-lived * Declaring a message queue twice is not an error; if parameters differ, new values are used if possible * Exclusive message queues are for advanced usages and often aren’t needed */
    • AMQP: Introducing Users A Connection can use several StormMQ enforces that the same authentication mechanisms Users and Permissions are in which are per Virtual Host all of a System’s Environments but but Commonest is User - Password Passwords are different and and Users are intended to be system Passwords are securely generated accounts (robots), not sysadmins by StormMQ and are 512 bit keys Users have read, write and create Configuration Managers can prevent permissions using a wildcard syntax: revelation of passwords to different important to name queues, etc well system users, eg developers vs sysadmins Aim: Separation of Responsibility Aim: Separation of Environments
    • / Example of create-system.json used with API call stormmq-create-system / { "companyName" : "usuallyYourUserName", "systemName" : "anythingAlphabetic", "environments" : [{ "environmentName" : "development", / Different StormMQ clusters have different characteristics (and costs) / "clusterName" : "free-1", / These are the StormMQ user accounts to which passwords for AMQP users (below) can be revealed for / this environment (development) "permittedStormMQUserNames" : [ "developerPeter", "developerJames", "developerJohn" ] }], / User permissions operate on Message Queues and Exchanges / "amqpUserPermissions" : { "queueReader" : { "create" : "^$", "read" : ".*", "write" : "^$" }, "queueWriter" : { "create" : "^$", "read" : "^$", "write" : ".*" }, "queueCreator" : { "create" : ".*", "read" : "^$", "write" : "^$" }, "onlyProcessingObjects" : { "create" : "^processing.*", "read" : "^processing.*", "write" : "^processing.*" } } }
    • Configuring AMQP Programmatic Config AMQP StormMQ Message Queues ✓ Active ✓ Same Exchanges ✓ Active ✓ Same Bindings ✓ Active ✓ Same Users ! Extra ✓ REST User Permissions ! Extra ✓ REST Entire Systems ! Extra ✓ REST* “*Our REST API has Java, Ruby, PHP and .NET bindings”
    • AMQP: Getting Statistics Programmatic Info AMQP StormMQ Message Queues ! Extra ✓ REST Exchanges ! Extra ✓ REST Bindings ! Extra ✓ REST Entire Systems ! Extra ✓ REST Usage Patterns ! Extra ✓ REST Generated Passwords ! Extra ✓ REST* “*Password revelation is restricted to specific individuals”
    • / Getting Statistics from StormMQ using the REST API in Java / / If you need to use proxies or experience the pain of Java SSL choose a different method in ApiConfiguration / final ApiConfiguration apiConfiguration = directConnectionIfHavingProblemsWithSslVerification(); / If your Magic Secret Key is in an unusual place choose a different method in Api / final Api api = apiByGuessingUsingOsConfigurationFiles( apiConfiguration, stormMqUserName ); final List<AmqpQueue> amqpQueues = api . listQueues( companyName, systemName, environmentName ); final List<AmqpExchange> amqpExchanges = api . listExchanges( companyName, systemName, environmentName ); final List<AmqpBinding> amqpBindings = api . listBindings( companyName, systemName, environmentName ); final Map<AmqpUserName, AmqpUserPassword> passwords = api . listAmqpUserNamesAndPasswords( companyName, systemName, environmentName ); final CompanyDetails companyDetails = api . describeCompany( companyName ); / Finding “what we’ve got” / final List<CompanyName> companyNames = api . listCompanies(); final List<ClusterName> clusterNames = api . listClusters( companyName ); final List<SystemName> systemNames = api . listSystems( companyName ); final SystemDetails systemDetails = api . describeSystem( companyName, systemName ); final Set<Environment> environments = systemDetails . getEnvironments(); /* Notes * Also consider the command-line tools stormmq-list-queues, stormmq-list-exchanges and stormmq-list-bindings * Further examples are in com.stormmq.api.Example (part of the url signer jar) */
    • Virtual Hosts Virtual Hosts: Think Apache! Virtual Hosts: StormMQ / Widgets Ltd / Invoicing System / Development Testing Production Virtual Hosts isolate Message Queues Virtual Hosts isolate Your Company Virtual Hosts isolate User Groups* Virtual Hosts isolate Your Systems Virtual Hosts isolate Exchanges* Virtual Hosts isolate Your Environments Virtual Hosts isolate Bindings* Virtual Hosts stop Data Accidents Virtual Hosts isolate Enable Configuration Management
    • Message Properties Persistent = 1 Server Priority = 0 Understood Good Practice Timestamp = 4568965345 MessageId = 567A-GH-20100709 Persistence is per-Message MIME Content-Type Priority Queues are per-Message Unique MessageId Timestamps expire Messages Schema & Version Control life of a Message Completely Optional “This is just the tip of messaging best practice with AMQP”
    • / Message Properties: Examples of Good Practice / / Being explicit is good practice / final BasicProperties messageHeaders = new BasicProperties(); {{ DeliveryMode . NonPersistent . setOn( this ); Priority . Zero . setOn( this ); setMessageId( "My unique ID, eg a GUID or ascending value" ); / Explicit MIME type including charset; use application/octect-stream for trully binary data / setContentType("application/json; charset=utf-8"); / Use compression for larger textual data; use the same values as HTTP headers; use "identity" for none / setContentEncoding( "gzip" ); / Timestamp each message / setTimestamp( new Date( System . currentTimeMillis() ) ); / Schema and version, perhaps using a XML namespace definition / setType( "syslog-message_1.0.4" ); / Optional: Use an expiration to discard old messages that are stale (milliseconds, example is 5 minutes) / setExpiration( Integer . toString( 5 * 60 * 1000) ); / Optional: Identify the sender for later debugging / setAppId( "sending application name" ); setUserId( "role of sender, or, user of machine" ); / Optional: Identify domain or environment (consumer could error if working in different one) / setClusterId( "production" ); }};
    • And there is more! Fine-Grained User Permissions Immediate Delivery More Message Properties Additional Exchange Queue Types Purging Custom Message QoS Properties Transactions Auto Deletion
    • Useful Book: Enterprise Integration Patterns Hohpe & Woolf