Messaging in the Cloud - AMQP, RabbitMQ and Spring


Published on

Shows how AMQP and RabbitMQ messaging work with Spring.

Published in: Technology

Messaging in the Cloud - AMQP, RabbitMQ and Spring

  1. 1. Messaging in the Cloud AMQP, RabbitMQ and Spring CONFIDENTIAL © 2010 SpringSource, A division of VMware. All rights reserved
  2. 2. Messaging in the Cloud   Need new levels of scalability   Need a standardized wire protocol •  Won't ship a specific client for each environment •  Better interoperability   JMS only defines an API and behavior, but no protocol   AMQP defines a protocol 2
  3. 3. Why AMQP?   Interoperability – like TCP and unlike JMS •  RabbitMQ leading the 2009 interoperability push around AMQP 0-9-1 towards AMQP 1.0   Multiple vendors on one open, royalty-free standard •  You are not locked in •  Lower risk, lower price because of competition, easier to compare •  Products specialized around different areas of value e.g low latency, high stability, wide area   Efficient – designed for today’s pubsub and queueing needs •  Binary wire protocol •  Support in all major languages •  Supported on most OS platforms   Already in use by many major companies   Future proof – backed by Cisco, Microsoft, VMware and many more 3
  4. 4. Broad platform and vendor support... 4
  5. 5. RabbitMQ is used a lot in the Amazon Cloud   RabbitMQ preferred to Amazon SQS 5
  6. 6. Key AMQP messaging protocol requirements   Internet protocol - like HTTP, TCP - but ASYNCHRONOUS   Ubiquity: Open, easy & low barrier to use, understand and implement   Safety: Secure and trusted global transaction network   Fidelity: Well-stated message queuing, ordering and delivery semantics   Applicability: any broker can talk to any client, support common messaging pattern and topologies   Interoperability   Manageability: Binary, scalable 6
  7. 7. AMQP  in  a  nutshell   Different languages Ruby, Java … C P X C P X C AMQP AMQP protocol protocol 7
  8. 8. Exchange Types: Matching Algorithms   Direct •  Routes on a routing key •  Two direct exchanges always exist • and the default exchange (with no public name) are mandatory   Topic •  Routes on a routing pattern • is mandatory if the server supports topic exchanges •  Which it should according to the spec (whereas direct and fanout must be supported)   Fanout •  Simple broadcast to all bound queues (no args when binding). Fast •  amq.fanout is mandatory Any Exchange that routes Messages to more than one queue will create multiple instances of the Message. 8
  9. 9. AMQP in more detail Messages are stateless C P X C P X C 9
  10. 10. AMQP in more detail Queues buffer messages for Exchanges are push to consumers X stateless routing tables. Queues are stateful, ordered, and can be persistent, transient, private, shared. Order might change if messages are redelivered C P X C P X C 10
  11. 11. AMQP in more detail Queues are bound to named exchanges Binding can have a pattern e.g. “tony” (direct exchange) or “*.ibm.*” (topic exchange) C P X C P X C 11
  12. 12. AMQP in more detail P Producers send messages to exchanges with routing key e.g. “tony”, or ordered set of keys e.g. “” Exchanges route messages to queues whose binding X pattern matches the message routing key or keys C P X C P X C 12
  13. 13. Twi1er  style  pubsub  message  flow   C Anders is at P X work is at work Tony P X is at is at is at work work work C Evan Evan and Anders want to follow what Tony says: •  bind queues to a RabbitMQ exchange •  pattern “tony”. Tony publishes “is at work” to exchange using routing key “tony”. Exchange updates Evan’s and Anders’ queues Other patterns are possible e.g. for filtering by topic similar to this: 13
  14. 14. Producers  and  consumers  logically  interact  through  a  broker  cloud   C P C P P X X P P P C C 14
  15. 15. RabbitMQ and Spring CONFIDENTIAL © 2010 SpringSource, A division of VMware. All rights reserved
  16. 16. Configuring Rabbit Resources with Spring   Spring enables decoupling of your application code from the underlying infrastructure   The container provides the resources   The application is simply coded against the API 16
  17. 17. Configuring a ConnectionFactory <bean id="connectionFactory" class="org.sfw.amqp.rabbit.connection.CachingConnectionFactory"> <property name="username" value="guest"/> <property name="password" value="guest"/> <property name="channelCacheSize" value="42"/> <property name="hostName" value="localhost"/> </bean> Caches connection i.e. connection has to be stateless i.e. can only be used for transactional access only (connection is stateful) Otherwise use com.rabbitmq.client.ConnectionFactory 17
  18. 18. Queues in RabbitMQ   Queue deliver messages to at max one consumer   Messages are sent to an exchange and can be routed to one or multiple queues   Meta data about RabbitMQ Queues is stored in org.springframework.amqp.core.Queue: •  String name •  boolean durable •  boolean exclusive : private to one consumer •  boolean autoDelete   Meta data can be used with RabbitAdminTemplate •  Call declare() to actually start using the Queue   Afterwards they are identified by name   Queues are bound to exchanges with routing keys   Default: Bound using the name of the queues as routing key 18
  19. 19. Exchanges in RabbitMQ   Meta data about RabbitMQ Exchanges is stored •  DirectExchange : String as routing key, Queue binds to exchange with key •  FanoutExchange : No routing, what goes in must go out •  TopicExchange : Pattern as routing key   String name   boolean durable   boolean autoDelete   Each message received by an exchange will be delivered to each (qualifying) Queue bound to it 19
  20. 20. Spring’s Templates   AmqpTemplate: Generic AMQP interface   RabbitOperations: Rabbit specific interface: (adds just a callback)   RabbitTemplate: Implementation   Spring might provide support for other AMQP implementations later   Common interface 20
  21. 21. Spring’s Templates   Central point to send and receive messages   Manages resources transparently   Throws runtime exceptions   Provides convenience methods and callbacks public Message receive() public Message receive(final String queueName) public void send(MessageCreator messageCreator) public void send(String routingKey, MessageCreator messageCreator) public void send(String exchange, String routingKey, MessageCreator messageCreator) 21
  22. 22. MessageConverter   The RabbitTemplate uses a MessageConverter to convert between objects and messages   The default SimpleMessageConverter handles basic types •  byte[] directly transfered •  String converted to byte[] •  Serializable serialized to byte[] •  Content type set accordingly   JsonMessageConverter converts from / to JSON using Jackson   MarshallingMessageConverter converts from / to XML using Spring's OXM mapping 22
  23. 23. Defining a RabbitTemplate Bean   Provide a reference to the ConnectionFactory   Optionally provide other references •  MessageConverter •  Routing key and exchange to be used if none is specified <bean id=“rabbitTemplate” class=“org.springframework.amqp.rabbit.core.RabbitTemplate”> <property name=“connectionFactory” ref=“connectionFactory”/> <property name=“messageConverter” ref=“messageConverter”/> <property name=“routingKey” value=“app.stock.request”/> </bean> 23
  24. 24. Sending Messages   The template provides options •  One line methods that leverage the template’s MessageConverter •  Callback-accepting methods that offer more flexibility   Use the simplest option for the task at hand 24
  25. 25. Sending Messages with Conversion   Leveraging the template’s MessageConverter public void convertAndSend(Object object); public void convertAndSend(String routingKey, Object object); public void convertAndSend(String exchange, String routingKey, Object object); 25
  26. 26. Sending Messages with Callbacks   When more control is needed, use callbacks public void convertAndSend(String routingKey, Object message, MessagePostProcessor messagePostProcessor); public void send(MessageCreator messageCreator); public void send(String routingKey, MessageCreator messageCreator); public void send(String exchange, String routingKey, MessageCreator messageCreator); Message createMessage() {…} 26
  27. 27. Setting the reply to / correlation ID   Allows request / reply schema i.e. wait for the reply to the specific message   Planned: sendAndReceive() as a direct implementation of this pattern getRabbitTemplate().convertAndSend(tradeRequest, new MessagePostProcessor() { public Message postProcessMessage(Message message) throws AmqpException { message.getMessageProperties().setReplyTo( new Address(defaultReplyToQueue)); try { message.getMessageProperties().setCorrelationId( UUID.randomUUID().toString().getBytes("UTF-8")); } catch (UnsupportedEncodingException e) { throw new AmqpExcpetion(e); } return message; } 27 });
  28. 28. Synchronous Message Reception   The RabbitTemplate can receive messages also •  receive() : Message •  receive(String queueName) •  receiveAndConvert() •  receiveAndConvert(String queueName)   If no message is on the queue null is returned   If no queueName is provided the queue name or queue set at the template will be used   The MessageConverter can be leveraged for message reception as well Object someSerializable = rabbitTemplate.receiveAndConvert(); 28
  29. 29. The MessageListener   The API defines this interface for asynchronous reception of messages public void onMessage(Message) { // handle the message } 29
  30. 30. Spring’s MessageListener Containers   Spring provides lightweight containers to call MessageListeners   SimpleMessageListenerContainer   Advanced scheduling and endpoint management options available 30
  31. 31. Defining a plain Message Listener <bean id="messageListenerContainer" class="org.sfw.amqp.rabbit.listener.SimpleMessageListenerContainer"> <property name="connectionFactory" ref="connectionFactory" /> <property name="queueName" value="#{},#{}"/> <property name="concurrentConsumers" value="5" /> <property name="messageListener" ref="messageListenerAdapter" /> </bean> 31
  32. 32. Spring's message-driven objects   Spring also allows you to specify a plain Java object that can serve as a listener <bean id="messageListenerAdapter" class="org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter"> <property name="delegate" ref="clientHandler" /> <property name="messageConverter" ref="jsonMessageConverter" /> </bean>   Parameter is automatically converted using a MessageConverter   Return value sent to response-destination or the reply to   Method with matching parameters is automatically called 32
  33. 33. Demo: Hello World   Producer send message using the RabbitTemplate   Routing key : helloWorldQueue name ""   Goes to Default Exchange   …and therefore to the helloWorldQueue (routing by name)   Consumer receives message using the default receive queue (helloWorldQueue) Routing key: P X C Default helloWorldQueue Exchange "" 33
  34. 34. Demo: Stock Exchange   Server sends stock prices   Client receives stock prices   Client can issue orders   Orders are processed by the server   Client receives confirmation of the trade   Queues are private for one client 34
  35. 35. Server sends stock prices Routing key: RabbitMarketDataGateway app.stockes.quote.?.? e.g. called periodicaly app.stockes.quote.nasdaq.ORCL P X Topic Exchange app.stock.marketdata 35
  36. 36. Client receives stock prices   Binding is created to attach the exchange to the queue using the routing key Routing key: app.stock.quotes.nasdaq.* X C Topic marketDataQueue Exchange (private per consumer) app.stock.marketdata ClientHandler in SimpleMessageListenerContainer with jsonMessageConverter and a MessageListenerAdapter 36
  37. 37. Client can issue orders   traderJoeQueue is set as reply to for the message Routing key: app.stock.request P X Default Exchange 37
  38. 38. Server processes order   Reply (i.e. confirmation) is sent to the reply to (trader Joe queue) X C Default stockRequestQueue ServerHandler in Exchange "app.stock.request" SimpleMessageListenerContainer with jsonMessageConverter and a MessageListenerAdapter 38
  39. 39. Client receives confirmation of the trade   Binding is created to attach the exchange to the queue using the routing key X C Default Exchange traderJoeQueue ClientHandler in SimpleMessageListenerContainer with jsonMessageConverter and a MessageListenerAdapter 39
  40. 40. Conclusion   Ubiquitous Messaging   AMQP: Protocol standard   Better scalability    Also a .NET version available 40