Successfully reported this slideshow.
Your SlideShare is downloading. ×

Advanced messaging with Apache ActiveMQ

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 51 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to Advanced messaging with Apache ActiveMQ (20)

Advertisement

Recently uploaded (20)

Advertisement

Advanced messaging with Apache ActiveMQ

  1. 1. Advanced Messaging with Apache ActiveMQ Bosanac Dejan May 2011 A Progress Software 1 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  2. 2. About me  Bosanac Dejan  Senior Software Engineer at FUSESource - http:// fusesource.com  Apache ActiveMQ committer and PMC member  Co-author of ActiveMQ in Action A Progress Software 2 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  3. 3. What we are going to cover?  What is ActiveMQ  The Basics  Flow control  Scaling  High Availability  Future  Conclusion A Progress Software 3 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  4. 4. What is ActiveMQ? A Progress Software 4 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  5. 5. Apache ActiveMQ  Apache ActiveMQ • Leading Open Source messaging platform • Supported Java Standards: o JMS 1.1, J2EE 1.4, JCA 1.5 and XA  Reliable, high performance messaging • Out-performs many legacy proprietary message queues • Configurable for many different deployments  Multi-Protocol/Multi-Language Support A Progress Software 5 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  6. 6. Background  ActiveMQ started in 2005 at CodeHaus  Moved to Apache Software Foundation in 2006  1,117,537 lines of code  24 committers  Now the most widely used open source messaging system on the planet A Progress Software 6 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  7. 7. The Basics A Progress Software 7 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  8. 8. Messaging is  Loosely coupled exchange of messages between applications  Location transparency  Can be persistent or non-persistent  Can be transactional A Progress Software 8 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  9. 9. Topics A Progress Software 9 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  10. 10. Topics  One message goes to 0-to-many consumers based on the current subscribers  Think like mailing lists or discussion forums  Ideal for publishing business events  Distributed observer pattern  Allows one part of your system to notify anyone else who may be interested in an event A Progress Software 10 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  11. 11. Queues A Progress Software 11 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  12. 12. Queues  Messages are load balanced across many consumers  Each message goes to exactly one consumer  Consumers compete for messages  Its easy to browse and monitor queues  Ideal for grid style applications A Progress Software 12 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  13. 13. Challanges  Create a general messaging platform  Support variety of use-cases • Large number of clients • Large number of destinations • Slow consumers  Provide enterprise feaures • Security • High availability • Management • etc A Progress Software 13 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  14. 14. Flow Control A Progress Software 14 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  15. 15. Flow Control - Why?  Dealing with deep queues  Dealing with slow consumers  We want to prevent broker from being flooded with messages  We want to prevent broker running out of memory and other resources A Progress Software 15 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  16. 16. Flow Control - How?  Message Cursors  Producer Flow Control A Progress Software 16 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  17. 17. Flow Control - Cursors A Progress Software 17 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  18. 18. Flow Control - Store-based Cursor A Progress Software 18 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  19. 19. Flow Control - Non-persistent cursor A Progress Software 19 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  20. 20. Flow Control - Limits Per destination <destinationPolicy> <policyMap> <policyEntries> <policyEntry queue=">" memoryLimit="10mb"/> </policyEntries> </policyMap> </destinationPolicy> System settings <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="256 mb" /> </memoryUsage> <storeUsage> <storeUsage limit="100 gb" /> </storeUsage> <tempUsage> <tempUsage limit="10 gb" /> </tempUsage> </systemUsage> </systemUsage> A Progress Software 20 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  21. 21. Flow Control - Producer Flow Control  Throttling producer speed to the speed of consumers A Progress Software 21 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  22. 22. Flow Control - VM Cursor <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"> <pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy> </policyEntry> A Progress Software 22 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  23. 23. Scaling A Progress Software 23 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  24. 24. Scaling - Types  Vertical scaling  Horizontal scaling  Traffic partitioning A Progress Software 24 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  25. 25. Vertical Scaling Increase load capacity using a single broker can handle. Problems: • Thread count • Memory usage • CPU usage A Progress Software 25 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  26. 26. Vertical Scaling - Threads What are threads used for • For Connections - Thread per Connection (blocking transport) • For Dispatching - Thread per Destination A Progress Software 26 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  27. 27. Vertical Scaling - Number of Connections Use non-blocking transport <transportConnectors> <transportConnector name="nio" uri="nio://0.0.0.0:61616"/> </<transportConnectors> Enables handling large number of clients A Progress Software 27 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  28. 28. Vertical Scaling - Number of Destinations Don’t use dedicated task runner ACTIVEMQ_OPTS="-Dorg.apache.activemq.UseDedicatedTaskRunner=false" Use optimized dispatch for queues <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" optimizedDispatch="true"> ... </policyEntry> </policyEntries> </policyMap> </destinationPolicy> A Progress Software 28 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  29. 29. Vertical Scaling - Memory  Give broker enough memory ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"  Configure big enough memory usage <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="1024 mb" /> </memoryUsage> ... </systemUsage> </systemUsage> A Progress Software 29 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  30. 30. Vertical Scaling - CPU  Disable tight encoding It uses more CPU to create smaller packets uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)"; A Progress Software 30 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  31. 31. Vertical Scaling - Conclusion There is a limit to the scalability a single machine can give A Progress Software 31 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  32. 32. Horizontal Scaling Increase load capacity using networked brokers Concepts:  Network of Broker A Progress Software 32 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  33. 33. Horizontal Scaling - Network of Brokers A Progress Software 33 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  34. 34. Horizontal Scaling - NOB Usage Configuration <networkConnector name="broker1-broker2" uri="static:(tcp://broker2:61617)" dynamicOnly="true" prefetchSize="1000" conduitSubscriptions="true" decreaseNetworkConsumerPriority="true" suppressDuplicateTopicSubscriptions="true" networkTTL="3"> </networkConnector> Connecting failover://(tcp://broker1:61616,tcp://broker2:61616)?randomize=true A Progress Software 34 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  35. 35. Horizontal Scaling - Conclusion  More latency in processing messages  Beware of complex topologies A Progress Software 35 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  36. 36. Hybrid Scaling Partition traffic to more non-connected brokers A Progress Software 36 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  37. 37. Hybrid Scaling - Conclusion  Pros • You can use all the tuning techniques used in Vertical scaling • Have better Horizontal scaleability than using Network Of Brokers (Less broker cross talk)  Cons • Added complexity required on the end user Application A Progress Software 37 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  38. 38. High Availability A Progress Software 38 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  39. 39. High Availability  Pure Master/Slave  JDBC Master/Slave  Shared File System Master/Slave A Progress Software 39 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  40. 40. Pure Master-Slave A Progress Software 40 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  41. 41. Pure Master-Slave  Shared nothing  Fully replicated • All messages • All acknowledgements • All transactions  Slave does not start any transports or network connections A Progress Software 41 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  42. 42. JDBC Master-Slave A Progress Software 42 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  43. 43. JDBC Master-Slave  Extreme reliability – but not as fast  Recommended if already using an enterprise database  No restriction on number of slaves  Simple configuration A Progress Software 43 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  44. 44. Shared Storage Master-Slave A Progress Software 44 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  45. 45. Shared Storage Master-Slave  Recommended if you have a SAN  No restriction on number of slaves  Simple configuration  N.B. – ensure file locking works – and times out – NFSv4 good! A Progress Software 45 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  46. 46. Future A Progress Software 46 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. FuseSource Company A Progress Software Company
  47. 47. Future - ActiveMQ Apollo  http://activemq.apache.org/apollo  ActiveMQ 5.x reached scalability and performance limits with the current architecture  New broker core A Progress Software 47 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  48. 48. Future - ActiveMQ Apollo  Reactor Based Thread Model  Scala 2.8 Implementation  Protocol Agnostic  REST Based Management A Progress Software 48 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  49. 49. Future - ActiveMQ Apollo Performance http://hiramchirino.net/blog 10 producers/10 consumers single topic using Stomp 20 byte payload A Progress Software 49 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  50. 50. Conclusions  Dynamic community  Leading in terms of messaging innovation  Built for Enterprise  Scalable, Good Performance, Reliable A Progress Software 50 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company
  51. 51. Questions?  ActiveMQ Web sites: • http://activemq.apache.org/ • http://fusesource.com/products/enterprise-activemq/  Blog: • http://www.nighttale.net/  Twitter: • http://twitter.com/dejanb • http://twitter.com/fusenews A Progress Software 51 Copyright © 2011 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. Company

×