JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel


Published on

Scalable Messaging is the the need of the enterprise messaging infrastructure. JBossA-MQ is one of the leading products for scalable messaging. It also covers some common concepts beween Apache Active MQ and JBoss A-MQ . Apache Camel as a integration framework supports the enterprise messaging greately and the presentation also contains some of its areas of adjunction to Scalable messaging with queues and topics.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel

  1. 1. 1
  2. 2. Scalable messaging with JBoss A-MQ and Camel Bhavani Ananth @bhavani_an in.linkedin.com/in/bhavaniananth/ 2 Naveen Raj Balasubramaniam @Naveenrajbala in.linkedin.com/in/naveenraj/
  3. 3. Messaging And Integration JBoss A-MQ JBoss A-MQ – Protocols JBoss A-MQ – Message Stores Vertical and Horizontal Scalability 3 Camel in brief! Agenda
  4. 4. Messaging And Integration 4
  5. 5. Scalable Solution 5 Large Retail Industry Billing Go Online Vendor Inventory Warehouse Logistics Integration Infrastructure MessagingMessaging Messaging Messaging MessagingMessaging
  6. 6. Data Is Growing Every Second – Data Deluge More Data Due to Emergence of Internet Of Things Need For A Scalable Solution
  7. 7. What do we have! Red Hat JBoss A-MQ 7
  8. 8. JBoss A-MQ 8
  9. 9. What is JBoss A-MQ 9 JMS 1.1-compliant messaging system based on Apache Active MQ. Consists of a broker and client-side libraries enabling remote communication among distributed client applications.
  10. 10. Architecture and Features 10 JBoss-AMQ Apache Karaf + Fuse Fabric Monitoring and Management Apache Active MQ STOMP MQTT Open Wire AMQP NMS JMS
  11. 11. JBoss A-MQ – Protocols 11
  12. 12. Protocols Supported 12 Client API’s JMS NMS CMS Rest/Ajax Wire Protocols MQTT AMQP STOMP Open Wire Transports TCP/SSL HTTP/HTTPS WS/WSS
  13. 13. MQTT MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. URI- mqtt://Host:[Port]?transportOptions 13 Facebook Messenger Smart Santander Features Open Simple Works well in constrained networks
  14. 14. AMQP Features  Multi-Vendor interoperability  Highly reliable  Feature rich AMQP (Advanced Message Queueing Protocol) is an openly published wire specification for asynchronous messaging URI: “amqp:// Host:[Port]?transportOptions 14
  15. 15. STOMP STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol. URI: “stomp://localhost:61613” 15 Broker Producer STOMP Consumer (Ruby,PHP) TCP Features  Simple  Used in scripting languages
  16. 16. Openwire Features Default wire format used by ActiveMQ. Efficient Supports auto reconnect Supports client failover OpenWire is the cross language Wire Protocol to allow native access to ActiveMQ from a number of different languages and platforms 16
  17. 17. JBoss A-MQ – Message Stores 17
  18. 18. Message Stores KahaDB message store Distributed KahaDB message store LevelDB message store – the best in terms of scalability and performance Journaled JDBC adapter Non-journaled JDBC adapter 18
  19. 19. Message Broker Message Cursors Message Store and Message cursors in an Retail Infrastructure 19 Vendor Systems System A System B Inventory System Message Store App Queue 2 Dispatch queue Pending Cursor App Queue 1 App Queue 3 Queue impl details
  20. 20. Message Broker Message Cursors – Increase Scalability of Message Stores Store Based cursors for fast Consumers (Retail Scenario) 20 Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
  21. 21. Store Based cursors for slow Consumers Message Cursors – Increase Scalability of Message Stores Contd… 21 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
  22. 22. VMCursors Message Cursors – Increase Scalability of Message Stores Contd… <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"> <pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy> </policyEntry> 22 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
  23. 23. File Based Cursors •Does not scale well if the consumers are behind by a large margin •Used for non persistent messages Message Cursors – Increase Scalability of Message Stores Contd… 23 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages File Based Cursors Temporary Files Vendor Systems System A System B
  24. 24. Vertical and Horizontal Scalability 24
  25. 25. Vertical Scalability  Capacity of a single broker to support large numbers of connections from consumers and producers.  To Consider  Thread Count  CPU Usage  Memory Usage 25
  26. 26. Vertical Scalability – Reducing Thread Count NIO protocol <transportConnectors> <transportConnector name="nio" uri="nio://"/> </<transportConnectors> 26
  27. 27. JBoss A-MQ – Vertical Scalability – Reducing Thread Count Disable Dedicated Task Runner ACTIVEMQ_OPTS="-org.apache.activemq.UseDedicatedTaskRunner=false“ Use optimized dispatch for queues <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" optimizedDispatch="true"> ... </policyEntry> </policyEntries> </policyMap> </destinationPolicy> 27
  28. 28. Vertical Scalability – CPU Enable tightEncodingEnabled uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)"; Enabling compression tcp://hostA:61617?jms.useCompression=true 28
  29. 29. Vertical Scalability – Memory Allocate more memory to broker • pass the –Xmx option to the JVM. ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false“ Configure big enough memory usage <systemUsage> <memoryUsage> <memoryUsage limit="1024 mb" /> </memoryUsage> </systemUsage> 29
  30. 30. Horizontal Scalability – Network of Brokers Concepts Network Connector Transport Connector Store and Forward 30 Broker-1 Broker-2 Network Connector Producer Transport Connector Consumer Transport Connector
  31. 31. Retail Example – Hub and Spoke Topology 31
  32. 32. Store and Forward- Configuration Store Store Retail Outlet Ware house Fire wall <networkConnectors> <networkConnector uri="static:(tcp://warehouse:61617)" userName="admin" password="admin“ duplex=“true” conduitSubscriptions=“true” decreaseNetworkConsumerPriorit y=“false” /> </networkConnectors>
  33. 33. When there are a lot of producers Brokers, A, B, and C, accept connections from message producers. Brokers, X and Y, accept connections from message consumers. Concentrator Topology - Producer Load Balancing 33 A B C YX C1 C2 C3 C4
  34. 34. Some Other Topologies 34  Tree Topology Graph Topology Mesh Topology
  35. 35. Retail Example-High Availability Shared Store Retail Master Broker Retail Slave Broker Wareh ouse Slave Broker Wareh ouse Master Broker Shared Store Network Connection on Failure Network Connection <networkConnectors> <networkConnector uri="static:(tcp://retail- master:61617, tcp://retail- slave:61617)" </networkConnectors> <services> <masterConnector remoteURI=“tcp://retail- slave:62001” userName=“admin” password=“admin”/> </services>
  36. 36. Other Master Slave Topologies Master Broker Slave Broker Store Store Message Shared Nothing Master Slave Reliable Only one slave possible Downtime when master fails Client 36
  37. 37. Shared Storage Master-Slave Master Broker with file lock Slave Broker Shared File System or SAN Shared File system Master Slave Recommended if SAN is available Easy to configure No restriction on the slaves Client Slave Broker Slave polls for lock 37
  38. 38. Hybrid Scaling 38 Client Acc Payable Master Acc Payable Slave Accounts Payable Information Acc Receivabl e Master Acc Receivabl e Slave Accounts Receivable Information  Client Complexity  Less broker cross talk
  39. 39. Apache Camel A ride to solve your integration problems 39
  40. 40. Camel-What is it Open source integration framework Based on Enterprise Integration Patterns(EIPs) Supports more than 120+ components Supports DSL programming with Spring, Java, Scala and groovy Flexible deployment Can run standalone with maven 40 CBR Wire Tap TransformationEndpoint Translator
  41. 41. Camel-Architecture 41
  42. 42. Camel Coding Structure import org.apache.camel.builder.RouteBuilder; public class MyRouteBuilder extends RouteBuilder { public void configure() { from("file:SourceFolder?noop=true") .to("file:TargetFolder"); } } <camelContext xmlns="http://camel.apache.org/schema/spring"> <route id=“FileToFile"> <from uri="file:SourceFolder?noop=true""/> <to uri="mock:inside"/> </route> </camelContext> Java DSL Spring DSL 42 Read a file from SourceFolder and write it to TargetFolder
  43. 43. 43 <route> <from uri="activemq:JUDCon?destination.consumer.exclusive=true&amp;destinati on.consumer.prefetchSize=50"/> <to uri="mock:2014"/> </route> Routes Camel and Messaging Configuring the Destination Options at the Endpoint in the Routes
  44. 44. Camel and Messaging Contd… <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> <property name="brokerURL" value="tcp://localhost:61616" /> </bean> Beans 44 <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="configuration" ref="jmsConfig"/> <property name="transacted" value="true"/> <property name="cacheLevelName" value="CACHE_CONSUMER" /> </bean> <bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop"> <property name= "maxConnections" value="8" /> <property name="connectionFactory" ref="jmsConnectionFactory" /> </bean> <bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration"> <property name="connectionFactory" ref="pooledConnectionFactory"/> <property name="concurrentConsumers" value="10“ /> </bean>
  45. 45. Camel – Scalability Techniques 45 Throttle Load Balancing from("activemq:queueA").throttle(10). to("activemq:queueB") <from uri=" activemq:queueA " /> <throttle maximumRequestsPerPeriod="3“> <to uri=" activemq:queueB " /> from(" activemq:queueA "). loadBalance().roundRobin(). to(" activemq:queueB ", " activemq:queueC", " activemq:queueD"); <from uri=" activemq:queueA "/> <loadBalance> <roundRobin/> <to uri=" activemq:queueB"/> <to uri=" activemq:queueC "/> <to uri=" activemq:queueD "/> </loadBalance>
  46. 46. 46 Embedding Camel into Your Broker • Reduces serialization cost • Reduced networkWhy? • Using Interceptors to your brokerWhen? • Add the camel route to activemq.xmlHow?
  47. 47. Summary • Messaging and Integration • Significance of Asynchronous messaging • How scalability is important to solve large flow of data • Scalability – Vertical and Horizontal • What's camel and Integration 47
  48. 48. Useful Links References • http://camel.apache.org/architecture.html • http://camel.apache.org/jms.html • http://activemq.apache.org/broker-camel-component.html 48
  49. 49. Thank You Scale Your Enterprise ECO System 49 @bhavani_an @Naveenrajbala